1. Field of the Invention
The present invention relates to an image processing method, an image processing device, and a recording medium, and in particular, to an image processing method for extracting a region which is assumed to be a region corresponding to a main portion in an image, to an image processing device to which the image processing method may be applied, and to a recording medium on which is recorded a program for executing the image processing method at a computer.
2. Description of the Related Art
When viewing a photograph of a person, the region to which the most attention is paid is the face of the person. For example, in a case in which an original image recorded on a photographic film is exposed and recorded (recorded by plane exposure or scanning exposure) onto a recording material such as a photographic printing paper, it is desirable to control the exposure such that the color and density of a person""s face are appropriate. However, to realize such control of exposure, the color tint and the density of the region corresponding to the person""s face in the original image must be accurately sensed. Further, among various image processings which have been developed in order to improve the image quality of an image expressed by image data obtained by the reading of the image, there are special image processings (such as correction of the density only at certain portions or red-eye correction or the like) which are carried out only on a region corresponding to a person""s face within the image or only on a portion of such a region. In order to carry out such processings, the position and size of the region corresponding to a person""s face within the image must be sensed accurately.
There have been proposed various conventional techniques for extracting a region assumed to correspond to a main portion such as a person""s face within an image. For example, Japanese Patent Application Laid-Open (JP-A) No. 8-184925 discloses a face region extraction method in which, on the basis of image data, one configuration pattern of configuration patterns peculiar to portions of a person existing in an image (e.g., configuration patterns expressing the contour of a head portion, the contour of a face, the inner structure of a face, the contour of a torso, and the like) is searched for. In accordance with the size and orientation of a detected configuration pattern and the positional relationship between the face of a person and a predetermined portion of the person expressed by the detected configuration pattern, a region assumed to correspond to a person""s face is set, other configuration patterns different from the detected configuration pattern are searched for, and a region (face region), which is within the set region and confirmed as being appropriate as a person""s face and is assumed to correspond to a person""s face, is extracted.
However, among the various techniques which have been proposed, none can always precisely extract a region corresponding to a main portion such as a person""s face in an image, and there are cases when mistakes are made in extracting a region corresponding to a main region. When such mistakes are made, a drawback arises in that the results of image processing or exposure control carried out by using the results of extraction are inappropriate.
For example, in the case of an image in which a person is photographed by using a flash in a back-lit setting, if a high-brightness background portion within the image is erroneously extracted as a region corresponding to a person""s face, when the amount of exposure at the time of exposing and recording the image onto a recording material is controlled on the basis of the color and the density of the extracted region, the recorded image will be of poor image quality as the region corresponding to the person""s face will be colored-in black. Further, even in cases in which a special image processing is to be carried out only on the extracted region or a portion thereof, if the special processing is carried out only on a region which is different than a region which corresponds to a person""s face within the image, the image quality of the image expressed by the image data deteriorates.
The present invention was developed in light of the aforementioned circumstances, and an object thereof is to provide an image processing method, an image processing device and a recording medium in which effects due to errors in extracting a region assumed to correspond to a main portion within an image can be mitigated.
The present inventors selected an arbitrary extraction method from among various extraction methods for extracting a region assumed to correspond to a main portion such as a person""s face in an image. They repeatedly conducted experiments of extracting regions corresponding to main portions from a large number of images by using this selected extraction method, by varying the processing conditions of the extraction method (e.g., the threshold value for determining whether a region is a main portion) and by varying the extraction method itself. By comparing the regions extracted by these respective extractions, the present inventors discovered that, in most cases, the extracted regions which corresponded to or substantially corresponded to a main portion within the image overlapped other extracted regions and the degree of overlapping with other extracted regions was high. Thus, the present inventors hypothesized that the probability of an extracted region being a region corresponding to a main portion within an image is strongly correlated with the degree of overlapping with other extracted regions.
Among the various extraction methods, due to differences in the algorithms thereof, there are extraction methods in which there is no overlapping of extracted regions even in a case in which plural regions are extracted in a single extraction process, and there are extraction methods in which there is the possibility of overlapping of extracted regions occurring when plural regions are extracted in a single extraction process. (For example, in an algorithm which traces an edge in an image and extracts a region corresponding to a main portion, if edge tracing is carried out by using respectively different points as the starting points, different regions in which portions thereof overlap may be extracted as regions corresponding to main portions.) The present inventors confirmed that, even in the latter type of extraction methods, it is often the case that plural regions are extracted by a single extraction process, and extracted regions which overlap with another extracted region correspond or substantially correspond to a main region within the image, and, as mentioned above, the probability of an extracted region being a region corresponding to a main portion in an image is strongly correlated to the degree of overlapping with another extracted region.
On the basis of the above, the image processing method of the first aspect of the invention comprises the steps of: on the basis of image data, extracting candidate regions which are assumed to correspond to main portions in an image represented by the image data; determining, for each of candidate regions which overlap with another candidate region among the extracted candidate regions, a degree of overlapping with another candidate region; and for an extracted candidate region, evaluating the probability of being a region corresponding to a main portion in the image, in such a manner that, for a candidate region overlapping with another candidate region, an evaluation of the probability of being a region corresponding to a main portion within the image is higher than said evaluation for a candidate region not overlapping with another candidate region, and becomes higher as a degree of overlapping with another candidate region increases.
In the first aspect of the present invention, first, on the basis of the image data, a candidate region assumed to correspond to a main portion within the image expressed by the image data is extracted. The number of candidate regions to actually be extracted by this extraction depends on the contents of the image expressed by the image data, but in the present invention, it is desirable that plural candidate regions are extracted. Improving the probability of extracting plural candidate regions can be realized by, for example, extracting candidate regions plural times by using different extraction methods, or by extracting candidate regions plural times by using the same extraction method but by varying the processing conditions. Further, if an extraction method is used in which there is the possibility that there will be overlapping of each candidate region when plural candidate regions are extracted by one extraction processing, it suffices to carrying out extraction of the candidate regions a single time.
Among the extracted candidate regions, for those candidate regions which overlap with another candidate region in the image, a degree of overlapping with another candidate region is determined. The computation of the degree of overlapping can be carried out in a case in which, for example, plural candidate regions are extracted, and there exists a candidate region which overlaps with another candidate region.
As in the second aspect, for example, at least one of the following values may be used as the degree of overlapping: a value in which a distance between a pair of overlapping candidate regions (the distance between the respective centers of a pair of candidate regions, or the distance between representative points which represent positions of a pair of candidate regions) is normalized by using respective sizes of the pair of overlapping candidate regions as a standard; a value in which a surface area of an overlapped region of a pair of overlapping candidate regions is normalized by using respective sizes of the pair of candidate regions as a standard; and a value in which a size along a predetermined direction of an overlapped region of a pair of overlapping candidate regions is normalized by using as a standard respective sizes along the predetermined direction of the pair of overlapping candidate regions. Further, an average value or a weighted average value of two or more values among the respective values may be used. By using a value which is normalized in a manner such as described above, the degree of overlapping of the candidate region can be accurately digitized regardless of the sizes of the candidate regions. In a case in which the sizes of an overlapping pair of candidate regions are different, among the pair of values of the degrees of overlapping, which are both determined by using the sizes of the both candidate regions as a standard, it is preferable to use the value expressing the smaller degree of overlapping as the degree of overlapping.
In the first aspect, the probability that an extracted region is a region corresponding to a main portion within the image is evaluated in such a manner that, for a candidate region which overlaps with another candidate region, the evaluation of the probability of being a region corresponding to a main portion within the image is higher than the evaluation for a candidate region which does not overlap with another candidate region, and the evaluation becomes higher as the degree of overlapping with another candidate region increases. In this way, among the extracted candidate regions, candidate regions which are confirmed by the above described experiments as having a high probability of being a region corresponding to a main portion within the image, i.e., candidate regions which overlap with another candidate region and which have a relatively high degree of overlapping with another candidate region, have a higher evaluation of the probability of being a region corresponding to a main portion within the image than do other candidate regions.
The evaluation of an extracted candidate region can be carried out by, for example, setting an evaluation value which expresses by a number the evaluated level. The setting of the evaluation values may be carried out, for example, such that candidate regions which do not overlap with other candidate regions receive zero or a low value as their evaluation value, pairs of overlapping candidate regions receive evaluation values which increase as the degree of overlapping increases, and candidate regions which overlap with plural candidate regions receive an evaluation value which is the sum total of their evaluation values set in accordance with the degrees of overlapping with the respective overlapping candidate regions.
Examples of post-processings which use the results of the evaluation, for an extracted candidate region, of the probability of being a region corresponding to a main portion in an image include a processing for selecting from the respective candidate regions a candidate region having a high probability of being a region corresponding to a main portion as in the third aspect, or a processing for weighting the extracted candidate region (or candidate region selected from the extracted candidate regions) as in the fourth aspect. Even if, among the extracted candidate regions, candidate regions which are actually not main portions in the image are included due to mistakes in the extraction of regions assumed to be main portions in the image, by carrying out the evaluation relating to the first aspect, the effects caused by a candidate region which is not a main portion can be reduced in the post-processings on the basis of the results of evaluation.
For example, if a candidate region having a high probability of being a region corresponding to a main portion is to be selected, by selecting from the extracted candidate regions only those candidate regions whose evaluation of the probability as a region corresponding to a main portion is high, the candidate regions which are actually not main portions can be excluded. Further, for example, if weighting of the extracted candidate regions is carried out, the effects of the candidate regions which are actually not main portions can be mitigated by applying a large weight to a candidate region whose evaluation of the probability of being a region corresponding to a main portion is high and by applying a small weight to a candidate region whose evaluation of the probability of being a region corresponding to a main portion is low.
In this way, in accordance with the first aspect of the present invention, for an extracted candidate region, the probability of being a region corresponding to a main portion is evaluated on the basis of the existence of overlapping with other candidate regions and the degree of overlapping. Thus, on the basis of the results of evaluation, effects caused by erroneous extraction of regions assumed to correspond to main portions in an image can be reduced.
If a processing for selecting, from the extracted candidate regions, a candidate region having a high probability of being a region corresponding to a main portion is carried out as a post- processing by using the results of evaluation for an extracted candidate region, as in the third aspect, it is preferable to select from the extracted candidate regions a candidate region having a high probability of being a region corresponding to a main portion, by comparing an evaluation value, for the extracted candidate region, expressing the results of evaluation of the probability of being a region corresponding to main region, with a threshold value. In this way, even if candidate regions which actually are not main portions are included among the extracted candidate regions, such candidate regions can be eliminated on the basis of their evaluation values. Candidate regions having a high probability of being a region corresponding to a main portion can be accurately selected by an easy processing, and the standard for selection of the candidate region can be corrected by changing the threshold value.
As in the fourth aspect, the post-processing utilizing the results of evaluation for an extracted candidate region may be a processing for computing, as an image characteristic amount of the main portion, a weighted average of image characteristic amounts of extracted candidate regions or selected candidate regions, by weighting an extracted candidate region or a candidate region selected from the extracted candidate regions in accordance with the evaluation value expressing the results of evaluation of the probability of being a region corresponding to a main portion. In this way, even in a case in which a candidate region which is actually not a main portion is included among the extracted candidate regions, the weight for such a candidate region can be lowered on the basis of the evaluation value, and the image characteristic amount of the main portion can be accurately determined by a simple processing. In the fourth aspect, the selection of the candidate regions when candidate regions selected from the extracted candidate regions are used may be carried out in the same way as in the third aspect.
Various types of processings can be contemplated for the post-processings which are carried out after the evaluation, for an extracted candidate region, of the probability of being a region corresponding to a main portion. However, in a case in which a candidate region is extracted erroneously or a candidate region is mis-evaluated, the extent of the effect on the results of the post-processing differs greatly in accordance with the type of post-processing. Therefore, in the fifth aspect of the present invention, in the first aspect, after evaluation, for an extracted candidate region, of the probability of being a region corresponding to a main portion, when carrying out, on the basis of the results of evaluation, a predetermined image processing which includes at least one of selecting a candidate region having a high probability of being a region corresponding to a main portion and weighting an extracted candidate region, the standard for evaluation is changed or the standard for the selection or the weighting is change in accordance with the type of the predetermined image processing.
In accordance with the fifth aspect, for example, in a case in which the predetermined image processing is an image processing whose results of processing are greatly affected when a candidate region is erroneously extracted or a candidate region is mis-evaluated, for example, as in the sixth aspect, one of the following can be carried out: the standard of evaluation of the probability of being a region corresponding to a main portion is changed so as to be made more strict; the standard for the selection is changed such that only a candidate region evaluated as having a high probability of being a region corresponding to a main portion is selected; and a standard for weighting is changed such that a weight for a candidate region evaluated as having a low probability of being a region corresponding to a main portion is relatively small. In this way, in the post-processings, even though there is the possibility that some of the candidate regions corresponding to main portions will not be selected from the extracted candidate regions or that there will be drawbacks such as the weights therefor being too small, the results of processing of the predetermined processing can be prevented from being overly affected even when a candidate region is erroneously extracted or a candidate region is mis-evaluated.
For example, in a case in which the predetermined image processing is an image processing whose processing results are not greatly affected even if a candidate region is erroneously extracted or a candidate region is mis-evaluated, for example, as in the seventh aspect, one of the following can be carried out: the standard of evaluation of the probability of being a region corresponding to a main portion is changed so as to be made less strict; the standard for the selection is changed such that a candidate region evaluated as having a low probability of being a region corresponding to a main portion is also selected; and a standard for weighting is changed such that a weight for a candidate region evaluated as having a low probability of being a region corresponding to a main portion is relatively large. If a candidate region is erroneously extracted or a candidate region is mis-evaluated, in the post-processings, there is the possibility that candidate regions which are actually not main portions will be selected or that the weights therefor will be too large. However, the effects on the results of processing of the post-processing can be diminished. On the other hand, proper processing conditions can be obtained because there is a high possibility that all of the candidate regions corresponding to main portions among the extracted candidate regions will be selected and that large weights will be applied to candidate regions corresponding to main portions.
In this way, in accordance with the fifth aspect, when a predetermined image processing is carried out after the evaluation, for an extracted candidate region, of the probability of being a region corresponding to a main portion, the standard for evaluation is changed or the standard for selection or weighting is changed in accordance with the type of the predetermined image processing. Thus, appropriate processing results can always be obtained regardless of the type of the predetermined image processing.
For example, when an image in which a person is photographed in a backlit setting is recorded onto a recording material, if a high-brightness background portion within the image is erroneously judged to be a region corresponding to a main portion (a person""s face in this case) and the recording density is controlled on the basis of the density of this region, the quality of the recorded image will be extremely low. Further, for example, when an image whose background portions are low-brightness is recorded onto a recording material, if the low-brightness background portion is erroneously judged to be a region corresponding to a main portion (the face of a person photographed in a backlit setting without the use of a flash, or the like) and the recording density is controlled on the basis of the density of this region, the quality of the recorded image will be a extremely low in this case as well.
Thus, in the eighth aspect of the present invention, in the first aspect, for a candidate region whose density within the candidate region is higher than a first predetermined value or lower than a second predetermined value, the evaluation of the probability of being a region corresponding to a main portion is lowered, or the standard for selection at the time a candidate region having a high probability of being a region corresponding to a main portion is selected is made higher.
In the eighth aspect of the present invention, for candidate regions whose density within the candidate region is higher than a first predetermined value or lower than a second predetermined value, i.e., for candidate regions whose density within the candidate region is either extremely high or extremely low, the evaluation of the probability of being a region corresponding to a main portion is lowered, or the standard for selection at the time of selecting a candidate region having a high probability of being a region corresponding to a main portion is raised. Thus, when the evaluation of the probability of being a region corresponding to a main portion is lowered, a region, whose density is either extremely high or extremely low and which is erroneously extracted as a candidate region, can be prevented, in the post-processings, from being selected as a candidate region having a high probability of being a region corresponding to a main portion and from having a large weight applied thereto.
If a processing, which includes selecting a candidate region having a high probability of being a region corresponding to a main portion, is carried out as the post-processing, by raising the standard for selection of a candidate region having a high probability of being a region corresponding to a main portion, a region whose density is either extremely high or extremely low and which is erroneously extracted as a candidate region can be prevented from being selected as a candidate region having a high probability of being a region corresponding to a main portion. Accordingly, even in a case in which a region whose density is extremely low or extremely high is erroneously extracted as a candidate region, the processing results of the post-processings are not inappropriate.
An image processing device of the ninth aspect of the present invention comprises: extracting means for, on the basis of image data, extracting candidate regions which are assumed to correspond to main portions in an image represented by the image data; computing means for determining, for each of candidate regions which overlap with another candidate region among the extracted candidate regions, a degree of overlapping with another candidate region; and evaluating means for evaluating, for an extracted candidate region, the probability of being a region corresponding to a main portion in the image, in such a manner that, for a candidate region overlapping with another candidate region, an evaluation of the probability of being a region corresponding to a main portion within the image is higher than said evaluation for a candidate region not overlapping with another candidate region, and becomes higher as a degree of overlapping with another candidate region increases. Thus, in the same way as in the first aspect, effects due to mistakes in the extraction of regions assumed to correspond to main portions within an image can be reduced.
A tenth aspect of the present invention is a recording medium on which is stored a program for executing at a computer a processing comprising: a first step of, on the basis of image data, extracting candidate regions which are assumed to correspond to main portions in an image represented by the image data; a second step of determining, for each of candidate regions which overlap with another candidate region among the extracted candidate regions, a degree of overlapping with another candidate region; and a third step of, for an extracted candidate region, evaluating the probability of being a region corresponding to a main portion in the image, in such a manner that, for a candidate region overlapping with another candidate region, an evaluation of the probability of being a region corresponding to a main portion within the image is higher than said evaluation for a candidate region not overlapping with another candidate region, and becomes higher as a degree of overlapping with another candidate region increases.
In the recording medium of the tenth aspect is recorded a program for implementing at a computer a processing including the aforementioned first through third steps, i.e., a processing relating to the image processing method of the first aspect of the present invention. Thus, by a computer reading and executing the program recorded on the recording medium, in the same way as in the first aspect, effects due to mistaken extraction of a region assumed to correspond to a main portion within an image can be reduced.